Multi-scale Feature Extraction and Fusion for Online Knowledge Distillation

Online knowledge distillation conducts transfer among all student models to alleviate the reliance on pre-trained models. However, existing online methods rely heavily prediction distributions and neglect further exploration of representational knowledge. In this paper, we propose a novel Multi-scale Feature Extraction Fusion method (MFEF) for distillation, which comprises three key components: Extraction, Dual-attention Fusion, towards generating more informative feature maps distillation. The multi-scale extraction exploiting divide-and-concatenate in channel dimension is proposed improve representation ability maps. To obtain accurate information, design dual-attention strengthen important spatial regions adaptively. Moreover, aggregate fuse former processed via fusion assist training Extensive experiments CIFAR-10, CIFAR-100, CINIC-10 show that MFEF transfers beneficial outperforms alternative various network architectures.